A test framework is a system that provides utilities that enable users to build test cases on top of the test framework in order to test a target product. Typically, the test framework provides general-purpose functionality, such as logging, error checking, data generation, etc. Some test frameworks also include wrappers around a target product's features. Many custom test frameworks built by many independent engineers provide similar functionality.
Test frameworks are usually monolithic packages that provide wrappers around the functions of a specific product. In a large company, test development teams often do not share test frameworks and instead re-implement common functionality in many individual, and largely custom, test frameworks. On the other end of the spectrum, teams are sometimes forced to leverage a large, complex framework that does considerably more than they need it to do, simply because it offers them 80% of the functionality they do need. In such cases, teams may need to force fit the remaining 20% of the functionality that they need into a test framework that was not designed to provide that functionality.
Often test frameworks are built using assumptions about implementation details of the product to be tested that affect the code high up in the test architecture hierarchy. Such test frameworks typically must undergo heavy refactoring when the features, API or UI of the target product change. For example, in such test frameworks, tasks such as migrating the tests from using one network protocol or client package to another are very difficult to do.
While embodiments are described herein by way of example for several embodiments and illustrative drawings, those skilled in the art will recognize that the embodiments are not limited to the embodiments or drawings described. It should be understood, that the drawings and detailed description thereto are not intended to limit embodiments to the particular form disclosed, but on the contrary, the intention is to cover all modifications, equivalents and alternatives falling within the spirit and scope as defined by the appended claims. The headings used herein are for organizational purposes only and are not meant to be used to limit the scope of the description or the claims. As used throughout this application, the word “may” is used in a permissive sense (i.e., meaning having the potential to), rather than the mandatory sense (i.e., meaning must). Similarly, the words “include”, “including”, and “includes” mean including, but not limited to.